Systems and methods for optimizing a machine learning model

ABSTRACT

A system for optimizing a machine learning model. The machine learning model generates predictions based on at least one input feature vector, each input feature vector having one or more vector values; and an optimization module with a processor and an associated memory, the optimization module being configured to: create at least one slice of the predictions based on at least one vector value, determine at least one optimization metric of the slice that is based on at least a total number of predictions for the vector value, and optimize the machine learning model based on the optimization metric.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/212,202, filed Mar. 25, 2021, the contents of which are herebyincorporated by reference in its entirety.

FIELD

The present disclosure relates to systems and methods for optimizing amachine learning model.

BACKGROUND

Machine learning models can take inputs and make predictions ordecisions based on the inputs. This involves computers learning fromprovided data so that they can carry out certain tasks. Many knowntechniques can be used to develop training data and build a machinelearning model based on the training data. After a model is built, itcan be fine-tuned or optimized to improve its performance.

Known techniques to optimize a machine learning model utilize an overallaggregate performance and/or average performance metrics. With suchtechniques, however, it is difficult to identify issues (e.g. wrongpredictions) that are mainly responsible for affecting the model'soverall performance. There is a desire and need to overcome thesechallenges.

SUMMARY

The present disclosure provides systems and methods to overcome theaforementioned challenges. The disclosed systems and methods use groups(facets or slices) of predictions to troubleshoot issues disregarded byknown model optimization approaches that are global in nature.

In an example embodiment, a system for optimizing a machine learningmodel is disclosed. The system can include a machine learning model thatgenerates predictions based on at least one input feature vector, eachinput feature vector having one or more vector values. The system caninclude an optimization module with a processor and an associatedmemory. The optimization module can be configured to create at least oneslice of the predictions based on at least one vector value, determineat least one optimization metric of the slice that is based on at leasta total number of predictions for the vector value, and optimize themachine learning model based on the optimization metric.

In various example embodiments, the optimization metric may includeaccuracy that is determined based on a ratio of the sum of a number oftrue positives (TPs) and true negatives (TNs) and a sum of a number ofTPs, false positives (FPs), TNs and false negatives (FNs). Theoptimization metric may further include an accuracy volume score (AVS)that is determined based on multiplying a complement of the accuracywith a distribution count of the slice.

In various example embodiments, the optimization metric may includeprecision that is determined based on a ratio of a number of TPs and asum of a number of TPs and FPs. The optimization metric may furtherinclude a precision volume score (PVS) that is determined based onmultiplying a complement of the precision with a distribution count ofthe slice.

In various example embodiments, the optimization metric may includerecall that is determined based on a ratio of a number of TPs and a sumof a number of TPs and FNs, and F1 that is determined based on a ratioof a multiplication of recall and precision and a sum of recall andprecision. The optimization metric may include a F1 volume score (F1VS)that is determined based on multiplying a complement of the F1 with adistribution count of the slice.

In various example embodiments, the optimization metric may includerecall that is determined based on a ratio of a number of TPs and a sumof a number of TPs and FNs. The optimization metric may further includea recall volume score (RVS) that is determined based on multiplying acomplement of the recall with a distribution count of the slice.

In various example embodiments, the at least one slice may include afirst slice and a second slice such that the first slice's optimizationmetric is higher than the second slice's optimization metric, and theoptimization module can be configured to fix the first slice to optimizethe machine learning model.

In various example embodiments, the at least one slice may include afirst slice and a second slice such that the first slice's optimizationmetric is equal to the second slice's optimization metric, and theoptimization module can be configured to fix both the first slice andthe second slice to optimize the machine learning model.

In various example embodiments, the at least one slice includes multipleslices based on multiple vector values respectively, and theoptimization module can be configured to sort the multiple slices basedon their respective optimization metric.

In an example embodiment, a computer-implemented method for optimizing amachine learning model is disclosed. The method may include obtainingmultiple predictions from a machine learning model, the predictionsbeing based on at least one input feature vector, each input featurevector having one or more vector values; creating at least one slice ofthe predictions based on at least one vector value; determining at leastone optimization metric of the slice that is based on at least a totalnumber of predictions for the vector value; and optimizing the machinelearning model based on the optimization metric.

BRIEF DESCRIPTION OF DRAWINGS

Other objects and advantages of the present disclosure will becomeapparent to those skilled in the art upon reading the following detaileddescription of exemplary embodiments, in conjunction with theaccompanying drawings, in which like reference numerals have been usedto designate like elements, and in which:

FIG. 1 shows a system for optimizing a machine learning model accordingto an exemplary embodiment of the present disclosure;

FIG. 2 shows a diagram comparing predictions with latent truth dataaccording to an exemplary embodiment of the present disclosure;

FIGS. 3A and 3B show diagrams of slices based on different vector valuesaccording to an exemplary embodiment of the present disclosure;

FIG. 4 shows a diagram with certain input feature vectors according toan exemplary embodiment of the present disclosure;

FIG. 5 shows a diagram comparing predictions with latent truth dataaccording to an exemplary embodiment of the present disclosure;

FIGS. 6A-6C show histograms that visualize generation of an accuracyvolume score according to an exemplary embodiment of the presentdisclosure;

FIG. 7 shows a normalized Accuracy Volume Score (AVS) of slices ofvarious dimensions according to exemplary embodiment of the presentdisclosure;

FIG. 8 shows an example platform visualizing a combination ofperformance and volume according to an exemplary embodiment of thepresent disclosure;

FIG. 9 shows an example of sorting and indexing the prediction slicesbased on their respective AVS according to exemplary embodiment of thepresent disclosure;

FIG. 10 shows a flowchart for a method for optimizing the machinelearning model according to an exemplary embodiment of the presentdisclosure; and

FIG. 11 illustrates a machine configured to perform computing operationsaccording to an embodiment of the present disclosure.

DESCRIPTION

The disclosed techniques can be deployed to analyze a machine learningmodel where certain predictions or groups of predictions generated bythe model have underperformed. The techniques described herein canidentify and analyze such predictions to optimize the machine learningmodel.

FIG. 1 shows an example system 100 for optimizing a machine learningmodel. The system 100 may include a machine learning model 110 that cangenerate multiple predictions 115 based on at least one input featurevector 105. The input feature vector 105 can have one or more vectorvalues. The machine learning model 110 can be trained using a trainingdataset and one or more algorithms.

Input feature vector 105, as used herein, can be an individualmeasurable property or characteristic of a phenomenon being observed.For example, FIG. 2 shows an example diagram 200 with example inputfeature vectors 105 shown as ‘REGION’, ‘CHG AMOUNT’, ‘LAST CHG AMOUNT’and ‘MERCHANT TYPE’, each with multiple vector values. ‘REGION’ can havevalues of ‘CA’ and ‘DE’. ‘CHG AMOUNT’ can have values of ‘21,000’,‘4,000’, ‘100’ and ‘34,000’. ‘LAST CHG AMOUNT’ can have values of‘4,000’, ‘4,000’, ‘100’ and ‘100’. ‘MERCHANT TYPE’ can have values of‘PAWN’ and ‘GAS’.

Diagram 200 further shows multiple predictions 115 (Predictions 1, 2, 3and 4), such that each prediction can have values based on each inputfeature vector 105. For example, Prediction 1 has values ‘CA’, ‘34,000’,‘100’ and ‘PAWN’. Prediction 2 has values ‘CA’, ‘100’, ‘100’ and ‘GAS’.Prediction 3 has values ‘DE’, ‘4,000’, ‘4,000’ and ‘GAS’. Prediction 4has values ‘CA’, ‘21,000’, ‘4,000’ and ‘PAWN’.

The system 100 can include an optimization module 120 with a processor122 (e.g., a central processing unit (CPU), a graphics processing unit(GPU), etc.) and an associated memory 124. The optimization module 120can be configured to create at least one slice (i.e. grouping) of thepredictions 115 based on at least one vector value 105.

In an example embodiment, user input (e.g. touchscreen, mouse-click,etc.) can be used to generate a slice by grouping the predictions 115 ona user interface. A machine learning algorithm can be applied to themultiple predictions 115 to create the at least one slice of thepredictions. As such, unsupervised learning algorithms (e.g. k-means)that do not require pre-existing labels can be used. Alternatelysupervised learning algorithms can also be used.

FIG. 3A shows an example slice based on REGION=CA created using any ofthe aforementioned techniques. Such a slice includes Predictions 4, 2and 1. FIG. 3B shows another example slice based on REGION=CA and CHGAMOUNT>20,000 created using any of the aforementioned techniques. Such aslice includes Predictions 4 and 1.

The optimization module 120 can be configured to determine at least oneoptimization metric of the slice that is based on at least a number oftotal predictions for the vector value. The determination of variousoptimization metrics are described as follows.

FIG. 4 shows an example diagram 400 to visualize a generation of variousoptimization metrics. Diagram 400 shows seven predictions (each inputfeature corresponding to a slice of prediction) compared with the latenttruth data (e.g. ground truth) to determine which of the multiplepredictions are correct or incorrect. Of these, for 410, 420, 430, 450and 470, the latent truth data matches the predictions. Therefore, theoverall accuracy is ( 5/7)*100=71.42%.

If a prediction of a slice is true (also called not false (NF)) and alatent truth of the slice is also true, their comparison is considered aTrue Positive (TP). 430 is an example of a TP. If a prediction is truebut a latent truth is false, their comparison is considered a FalsePositive (FP). 440 is an example of a FP. If the prediction is false anda latent truth is also false, their comparison is considered a TrueNegative (TN). 410, 420, 450 and 470 are examples of a TN. If theprediction is false but a latent truth is true, their comparison isconsidered a False Negative (FN). 460 is an example of a FN. Therefore,the number of TPs=1, the number of FPs=1, the number of TNs=4 and thenumber of FNs=1.

In an example embodiment, the optimization module 120 can be configuredto determine an optimization metric called Accuracy, which can be aratio of the sum of a number of TPs and TNs and sum of a number of TPs,FPs, TNs and FNs. That is, Accuracy=(TP+TN)/(TP+FP+TN+FN). In theexample of diagram 400, Accuracy=(1+4)/(1+1+4+1)= 5/7.

In an example embodiment, the optimization module 120 can be configuredto determine an optimization metric called Precision, which can be aratio of a number of TPs and a sum of a number of TPs and FPs. That is,Precision=TP/(TP+FP). In the example of diagram 400,Precision=(1)/(1+1)=½.

In an example embodiment, the optimization module 120 can be configuredto determine an optimization metric called Recall, which can be a ratioof a number of TPs and a sum of a number of TPs and FNs. That is,Recall=TP/(TP+FN). In the example of diagram 400, Recall=(1)/(1+1)=½.

In an example embodiment, the optimization module 120 can be configuredto determine an optimization metric called F1, which can be a ratio of amultiplication of Recall and Precision and a sum of Recall andPrecision. That is, F1=Recall*Precision/(Recall+Precision). In theexample of diagram 400, F1=(½*½)/(½+½)=¼.

FIG. 5 shows another example diagram 500 to visualize a generation ofvarious optimization metrics. Diagram 500 illustrates four predictionscompared with the latent truth data. Of these, for 520 and 530, thelatent truth data matches the predictions. Therefore, the overallaccuracy is (½)*100=50%.

Predictions 4, 2 and 1 form a slice of prediction with REGION as CA. Ofthese, Prediction 4 is a FP (prediction is true but latent truth isfalse), Prediction 2 is a TP (prediction and latent truth are bothtrue), and Prediction 1 is a FP (prediction is true but latent truth isfalse). Therefore, for ‘REGION=CA’ slice of prediction,Accuracy=(TP+TN)/(TP+FP+TN+FN)=(1+0)/(1+2+0+0)=⅓. Prediction 3 forms aslice of prediction with REGION as DE. Prediction 3 is a TN (predictionand latent truth are both false. Therefore, for ‘REGION=DE’ slice ofprediction, Accuracy=(TP+TN)/(TP+FP+TN+FN)=(0+1)/(0+0+1+0)=1.

While the previous example is directed at the Accuracy optimizationmetric, a person of ordinary skill in the art would appreciate thatother features of the model can be analyzed. For example, optimizationmetrics such Precision, Recall, F1, False Positive Rate, False NegativeRate can be analyzed, as previously described. Similarly, otherclassification optimization metrics such as Sensitivity, Specificity canbe analyzed. Regression optimization metrics such as root-mean-squarederror (RMSE), mean-squared error (MSE), mean average error (MAE), meanaverage percentage error (MAPE), and % of predictions outside of %performance threshold can also be analyzed.

FIGS. 6A, 6B, 6C respectively show example histograms 610, 620 and 630to visualize a generation of various optimization metrics in accordancewith the disclosed principles. Histogram 610 shows a distribution countbased on various input feature vectors values in diagram 500. Thedistribution count of a slice can be the number of predictions of thatslice.

For example, the distribution count of slice ‘REGION=CA’ is 3. thedistribution count of slice ‘REGION=DE’ is 1. The distribution count ofslice ‘CHG AMOUNT=0-3,000’ is 1. The distribution count of slice ‘CHGAMOUNT=3,001-9,000’ is 1. The distribution count of slice ‘CHGAMOUNT=9,001-50,000’ is 2. The distribution count of slice ‘LAST CHGAMOUNT=0-3,000’ is 1. The distribution count of slice ‘LAST CHGAMOUNT=3,001-9,000’ is 1. The distribution count of slice ‘MERCHANTTYPE=PAWN’ is 2. The distribution count of slice ‘MERCHANT TYPE=GAS is2.

Histogram 620 shows the Accuracy of the various prediction slices. TheAccuracy of REGION=CA is 33% (or ⅓) and the accuracy of REGION=DE is100% (or 1), as previously described. Similarly, calculating based onthe previously described formula, Accuracy=(TP+TN)/(TP+FP+TN+FN), theAccuracy of CHG AMOUNT between 0-3,000 is 100% (or 1), CHG AMOUNTbetween 3,001-9,000 is 100% (or 1), and the Accuracy of CHG AMOUNTbetween 9,001-50,000 is 0. The Accuracy of LAST CHG AMOUNT between0-3,000 is 50% (or ½) and the Accuracy of LAST CHG AMOUNT between3,001-9,000 is also 50% (or ½). The Accuracy of MERCHANT TYPE=PAWN is 0and the Accuracy of MERCHANT TYPE=GAS is 100% (or 1).

In an example embodiment, to link performance and volume of a slice intoa single optimization metric, the performance by slice can be multipliedwith the volume of the slice. Such a metric provides a fair comparisonof slices irrespective of their size. This may allow for a creation ofcomplex multidimensional slices and use the same metric for performanceanalysis. By fixing/adjusting the slice with the highest score, theperformance of machine learning model can improve the most. Because thevolume is normalized as part of the score, comparison of small volumeslices to large volume slices can be made. This allows a creation ofcomplex multidimensional slices and use the same metric for performanceanalysis.

Histogram 630 shows an example of an optimization metric called AccuracyVolume Score (AVS) of the various slices in histograms 610 or 620. Theoptimization module 120 can be configured to generate AVS of the sliceby multiplying a complement of the Accuracy with the distribution count.That is, AVS equals (1−Accuracy)*Distribution count.

For REGION=CA, AVS=(1−⅓)*3=2. For REGION=DE, AVS=(1−1)*1=0. For CHGAMOUNT=0-3,000, AVS=(1−1)*1=0. For CHG AMOUNT=3,001-9,000,AVS=(1−1)*1=0. For CHG AMOUNT=9,001-50,000, AVS=(1−0)*2=2. For LAST CHGAMOUNT=0-3,000, AVS=(1−½)*1=½. For LAST CHG AMOUNT=3,001-9,000,AVS=(1−½)*1=½. For MERCHANT TYPE=PAWN, AVS=(1−0)*2=2. For MERCHANTTYPE=GAS, AVS=(1−1)*2=0.

FIG. 7 shows sorted AVS of slices of various dimensions. The dimensionof a slice depends on a number of input feature vectors used to createthe slice. For example, if the slice is based on only one input featurevector, it is considered a single dimensional slice. Similarly, if theslice is based on two input feature vectors, it is considered atwo-dimensional slice. Because the slices in FIG. 7 are normalized byvolume, various dimensions can be properly compared. Similarly, otheroptimization metrics can also be used for comparison.

FIG. 8 shows an example platform visualizing a combination ofperformance and volume based on the Accuracy optimization metric. Thehighlighted slices are one standard deviation from average performance.Of course, platforms based on other optimization metrics can also begenerated.

In various example embodiments, the optimization module 120 can beconfigured to generate an optimization metric called Recall Volume Score(RVS) of the slice by multiplying a complement of the Recall with thedistribution count. That is, RVS may equal (1-Recall)*Distributioncount. The optimization module 120 can be configured to generate anoptimization metric called Precision Volume Score (PVS) of the slice bymultiplying a complement of the Precision with the distribution count.That is, PVS may equal (1-Precision)*Distribution count. Theoptimization module 120 can be configured to generate an optimizationmetric called F1 Volume Score (F1VS) of the slice by multiplying acomplement of the F1 with the distribution count. That is, F1VS mayequal (1−F1)*Distribution count.

Similarly, in various example embodiments, the optimization module 120can be configured to generate an optimization metric called MAE VolumeScore (MAEVS) of the slice by multiplying MAE with the distributioncount. That is, MAEVS may equal MAE*Distribution count. The optimizationmodule 120 can be configured to generate an optimization metric calledMAPE Volume Score (MAPEVS) of the slice by multiplying MAPE with thedistribution count. That is, MAPEVS may equal MAPE*Distribution count.The optimization module 120 can be configured to generate anoptimization metric called RMSE Volume Score (RMSEVS) of the slice bymultiplying RMSE with the distribution count. That is, MAEVS may equalRMSE*Distribution count. The optimization module 120 can be configuredto generate an optimization metric called MSE Volume Score (MSEVS) ofthe slice by multiplying MSE with the distribution count. That is, MSEVSmay equal MSE*Distribution count.

In an example embodiment, the optimization module 120 can be configuredto generate various optimization metrics based on a Slice Fraction of aslice, which is a ratio of the number of predictions of that slice and atotal number of predictions. For example, in the example of FIGS. 3A and3B, the Slice Fraction of slice ‘REGION=CA’ is ¾ and the Slice Fractionof slice ‘REGION=CA & CHG AMOUNT>20,000’ is 2/4.

After obtaining the Slice Fraction, in various example embodiments, theoptimization module 120 can be configured to generate an optimizationmetric called Volume Score Accuracy (Error Contribution) by multiplyinga complement of the Accuracy with the Slice Fraction. That is, VolumeScore Accuracy (Error Contribution) may equal (1−Accuracy)*SliceFraction. The optimization module 120 can be configured to generate anoptimization metric called Volume Score Recall (Error Contribution) bymultiplying a complement of the Recall with the Slice Fraction. That is,Volume Score Recall (Error Contribution) may equal (1-Recall)*SliceFraction. The optimization module 120 can be configured to generate anoptimization metric called Volume Score Precision (Error Contribution)by multiplying a complement of the Precision with the Slice Fraction.That is, Volume Score Precision (Error Contribution) may equal(1-Precision)*Slice Fraction. The optimization module 120 can beconfigured to generate an optimization metric called Volume Score F1(Error Contribution) by multiplying a complement of the F1 with theSlice Fraction. That is, Volume Score F1 (Error Contribution) may equal(1−F1)*Slice Fraction.

Similarly, in various example embodiments, the optimization module 120can be configured to generate an optimization metric called Volume ScoreMAE (Error Contribution) by multiplying MAE with the Slice Fraction.That is, MAEVS may equal MAE*Slice Fraction. The optimization module 120can be configured to generate an optimization metric called Volume ScoreMAPE (Error Contribution) by multiplying MAPE with the Slice Fraction.That is, MAPEVS may equal MAPE*Slice Fraction. The optimization module120 can be configured to generate an optimization metric called VolumeScore RMSE (Error Contribution) by multiplying RMSE with the SliceFraction. That is, MAEVS may equal RMSE*Slice Fraction. The optimizationmodule 120 can be configured to generate an optimization metric calledVolume Score RMSE (Error Contribution) by multiplying MSE with the SliceFraction. That is, MSEVS may equal MSE*Slice Fraction.

In an example embodiment, the optimization module 120 can be configuredto create a first slice and a second slice that are based on a firstvector value and a second vector value, respectively, and compare theaccuracy volume score of the first slice with that of the second slice.For example, the first slice can be based on ‘REGION=CA’ and the secondslice can be based on ‘REGION=DE’, with their accuracy volume scorebeing 2 and 0, respectively. The machine learning model can then beoptimized by fixing the ‘REGION=CA’ slice because it has a higheraccuracy volume score. Of course, if the user so chooses, he/she canoptimize the machine learning model by fixing the ‘REGION=DE’ slice.

In an exemplary embodiment, if the accuracy volume score of the firstslice and the second slice are the same, the machine learning model canbe optimized by fixing both slices. For example, the first slice can bebased on ‘REGION=CA’ and the second slice can be based on ‘CHGAMOUNT=9,0001-50,000, with their accuracy volume score being 2 each. Themachine learning model can be optimized by fixing both slices. Ofcourse, if the user so chooses, he/she can optimize the machine learningmodel by fixing any one of these slices.

FIG. 9 shows an example of the optimization module 120 being configuredto sort and index the prediction slices based on their respective AVSaccording to an example embodiment. Of course, the sorting and index arenot limited by AVS and can be done based on various optimizationmetrics. Known techniques for sorting and indexing can be used. This canallow for fast searching and finding.

FIG. 10 shows a flowchart of a method 1000 for optimizing a machinelearning model according to an embodiment of the present disclosure. Themethod 1000 can include a step 1010 of obtaining multiple predictionsfrom a machine learning model such that the predictions are based on atleast one input feature vector, each input feature vector having one ormore vector values. Aspects of step 1010 relate to the previouslydescribed machine learning model 110 of the system 100.

The method 1000 can include a step 1020 of creating at least one sliceof the predictions based on at least one vector value, a step 1030 ofdetermining at least one optimization metric of the slice that is basedon at least a total number of predictions for the vector value, and astep 1040 of optimizing the machine learning model based on theoptimization metric. Aspects of the steps 1020, 1030 and 1040 relate tothe previously described optimization module 120 of the system 100.

FIG. 11 is a block diagram illustrating an example computer system 1100upon which any one or more of the methodologies (e.g. system 100 and/ormethod 1000) herein discussed may be run according to an exampledescribed herein. Computer system 1100 may be embodied as a computingdevice, providing operations of the components featured in the variousfigures, including components of the system 100, method 1000, or anyother processing or computing platform or component described orreferred to herein.

In alternative embodiments, the computer system 1100 can operate as astandalone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the computing system 1100 mayoperate in the capacity of either a server or a client machine inserver-client network environments, or it may act as a peer machine inpeer-to-peer (or distributed) network environments.

Example computer system 1100 includes a processor 1102 (e.g., a centralprocessing unit (CPU), a graphics processing unit (GPU) or both), a mainmemory 1104 and a static memory 1106, which communicate with each othervia an interconnect 1108 (e.g., a link, a bus, etc.). The computersystem 1100 may further include a video display unit 1110, an inputdevice 1112 (e.g. keyboard) and a user interface (UI) navigation device1114 (e.g., a mouse). In one embodiment, the video display unit 1110,input device 1112 and UI navigation device 1114 are a touch screendisplay. The computer system 1100 may additionally include a storagedevice 1116 (e.g., a drive unit), a signal generation device 1118 (e.g.,a speaker), an output controller 1132, and a network interface device1120 (which may include or operably communicate with one or moreantennas 1130, transceivers, or other wireless communications hardware),and one or more sensors 1128.

The storage device 1116 includes a machine-readable medium 1122 on whichis stored one or more sets of data structures and instructions 1124(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1124 mayalso reside, completely or at least partially, within the main memory1104, static memory 1106, and/or within the processor 1102 duringexecution thereof by the computer system 1100, with the main memory1104, static memory 1106, and the processor 1102 constitutingmachine-readable media.

While the machine-readable medium 1122 (or computer-readable medium) isillustrated in an example embodiment to be a single medium, the term“machine-readable medium” may include a single medium or multiple medium(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more instructions 1124.

The term “machine-readable medium” shall also be taken to include anytangible medium that is capable of storing, encoding or carryinginstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present disclosureor that is capable of storing, encoding or carrying data structuresutilized by or associated with such instructions.

The term “machine-readable medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, optical media,magnetic media or other non-transitory media. Specific examples ofmachine-readable media include non-volatile memory, including, by way ofexample, semiconductor memory devices (e.g., Electrically ProgrammableRead-Only Memory (EPROM), Electrically Erasable Programmable Read-OnlyMemory (EEPROM)) and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

The instructions 1124 may further be transmitted or received over acommunications network 1126 using a transmission medium via the networkinterface device 1120 utilizing any one of several well-known transferprotocols (e.g., HTTP). Examples of communication networks include alocal area network (LAN), wide area network (WAN), the Internet, mobiletelephone networks, Plain Old Telephone (POTS) networks, and wirelessdata networks (e.g., Wi-Fi, 3G, and 4G LTE/LTE-A or WiMAX networks). Theterm “transmission medium” shall be taken to include any intangiblemedium that can store, encoding, or carrying instructions for executionby the machine, and includes digital or analog communications signals orother intangible medium to facilitate communication of such software.

Other applicable network configurations may be included within the scopeof the presently described communication networks. Although exampleswere provided with reference to a local area wireless networkconfiguration and a wide area Internet network connection, it will beunderstood that communications may also be facilitated using any numberof personal area networks, LANs, and WANs, using any combination ofwired or wireless transmission mediums.

The embodiments described above may be implemented in one or acombination of hardware, firmware, and software. For example, thefeatures in the system architecture 1100 of the processing system may beclient-operated software or be embodied on a server running an operatingsystem with software running thereon. While some embodiments describedherein illustrate only a single machine or device, the terms “system”,“machine”, or “device” shall also be taken to include any collection ofmachines or devices that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

Examples, as described herein, may include, or may operate on, logic orseveral components, modules, features, or mechanisms. Such items aretangible entities (e.g., hardware) capable of performing specifiedoperations and may be configured or arranged in a certain manner. In anexample, circuits may be arranged (e.g., internally or with respect toexternal entities such as other circuits) in a specified manner as amodule, component, or feature. In an example, the whole or part of oneor more computer systems (e.g., a standalone, client or server computersystem) or one or more hardware processors may be configured by firmwareor software (e.g., instructions, an application portion, or anapplication) as an item that operates to perform specified operations.In an example, the software may reside on a machine readable medium. Inan example, the software, when executed by underlying hardware, causesthe hardware to perform the specified operations.

Accordingly, such modules, components, and features are understood toencompass a tangible entity, be that an entity that is physicallyconstructed, specifically configured (e.g., hardwired), or temporarily(e.g., transitorily) configured (e.g., programmed) to operate in aspecified manner or to perform part or all operations described herein.Considering examples in which modules, components, and features aretemporarily configured, each of the items need not be instantiated atany one moment in time. For example, where the modules, components, andfeatures comprise a general-purpose hardware processor configured usingsoftware, the general-purpose hardware processor may be configured asrespective different items at different times. Software may accordinglyconfigure a hardware processor, for example, to constitute a particularitem at one instance of time and to constitute a different item at adifferent instance of time.

Additional examples of the presently described method (e.g. 1000),system (e.g. 100), and device embodiments are suggested according to thestructures and techniques described herein. Other non-limiting examplesmay be configured to operate separately or can be combined in anypermutation or combination with any one or more of the other examplesprovided above or throughout the present disclosure.

It will be appreciated by those skilled in the art that the presentdisclosure can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentlydisclosed embodiments are therefore considered in all respects to beillustrative and not restricted. The scope of the disclosure isindicated by the appended claims rather than the foregoing descriptionand all changes that come within the meaning and range and equivalencethereof are intended to be embraced therein.

It should be noted that the terms “including” and “comprising” should beinterpreted as meaning “including, but not limited to”. If not alreadyset forth explicitly in the claims, the term “a” should be interpretedas “at least one” and “the”, “said”, etc. should be interpreted as “theat least one”, “said at least one”, etc. Furthermore, it is theApplicant's intent that only claims that include the express language“means for” or “step for” be interpreted under 35 U.S.C. 112(f). Claimsthat do not expressly include the phrase “means for” or “step for” arenot to be interpreted under 35 U.S.C. 112(f).

We claim:
 1. A system for optimizing a machine learning model, thesystem comprising: a machine learning model that generates predictionsbased on at least one input feature vector, each input feature vectorhaving one or more vector values; and an optimization module with aprocessor and an associated memory, the optimization module beingconfigured to: create at least one slice of the predictions based on atleast one vector value, determine a mean average error volume score(MAEVS) of the slice based on a multiplication of a mean average error(MAE) with a distribution count of the slice, and optimize the machinelearning model based on the MAEVS.
 2. The system of claim 1, wherein theat least one slice includes a first slice and a second slice such thatthe first slice's MAEVS is higher than the second slice's MAEVS, and theoptimization module is further configured to fix the first slice tooptimize the machine learning model.
 3. The system of claim 1, whereinthe at least one slice includes a first slice and a second slice suchthat the first slice's MAEVS is equal to the second slice's MAEVS, andthe optimization module is further configured to fix both the firstslice and the second slice to optimize the machine learning model. 4.The system of claim 1, wherein the at least one slice includes multipleslices based on multiple vector values respectively, and theoptimization module is further configured to sort the multiple slicesbased on their respective MAEVS.
 5. A computer-implemented method foroptimizing a machine learning model, the method comprising: obtainingmultiple predictions from a machine learning model, the predictionsbeing based on at least one input feature vector, each input featurevector having one or more vector values; creating at least one slice ofthe predictions based on at least one vector value; determining a meanaverage error volume score (MAEVS) of the slice based on amultiplication of a mean average error (MAE) with a distribution countof the slice; and optimizing the machine learning model based on theMAEVS.
 6. The method of claim 5, wherein the at least one slice includesa first slice and a second slice such that the first slice's MAEVS ishigher than the second slice's MAEVS, and the optimizing includes fixingthe first slice to optimize the machine learning model.
 7. The method ofclaim 5, wherein the at least one slice includes a first slice and asecond slice such that the first slice's MAEVS is equal to the secondslice's MAEVS, and the optimizing includes fixing both the first sliceand the second slice to optimize the machine learning model.
 8. Themethod of claim 5, wherein the at least one slice includes multipleslices based on multiple vector values respectively, and the methodfurther comprises: sorting the multiple slices based on their respectiveMAEVS.
 9. A system for optimizing a machine learning model, the systemcomprising: a machine learning model that generates predictions based onat least one input feature vector, each input feature vector having oneor more vector values; and an optimization module with a processor andan associated memory, the optimization module being configured to:create at least one slice of the predictions based on at least onevector value, determine a mean average percentage error volume score(MAPEVS) of the slice based on a multiplication of a mean averagepercentage error (MAPE) with a distribution count of the slice, andoptimize the machine learning model based on the MAPEVS.
 10. The systemof claim 9, wherein the at least one slice includes a first slice and asecond slice such that the first slice's MAPEVS is higher than thesecond slice's MAPEVS, and the optimization module is further configuredto fix the first slice to optimize the machine learning model.
 11. Thesystem of claim 9, wherein the at least one slice includes a first sliceand a second slice such that the first slice's MAPEVS is equal to thesecond slice's MAPEVS, and the optimization module is further configuredto fix both the first slice and the second slice to optimize the machinelearning model.
 12. The system of claim 9, wherein the at least oneslice includes multiple slices based on multiple vector valuesrespectively, and the optimization module is further configured to sortthe multiple slices based on their respective MAPEVS.
 13. Acomputer-implemented method for optimizing a machine learning model, themethod comprising: obtaining multiple predictions from a machinelearning model, the predictions being based on at least one inputfeature vector, each input feature vector having one or more vectorvalues; creating at least one slice of the predictions based on at leastone vector value; determining a mean average error volume score (MAPEVS)of the slice based on a multiplication of a mean average percentageerror (MAPE) with a distribution count of the slice; and optimizing themachine learning model based on the MAPEVS.
 14. The method of claim 13,wherein the at least one slice includes a first slice and a second slicesuch that the first slice's MAPEVS is higher than the second slice'sMAPEVS, and the optimizing includes fixing the first slice to optimizethe machine learning model.
 15. The method of claim 13, wherein the atleast one slice includes a first slice and a second slice such that thefirst slice's MAPEVS is equal to the second slice's MAPEVS, and theoptimizing includes fixing both the first slice and the second slice tooptimize the machine learning model.
 16. The method of claim 13, whereinthe at least one slice includes multiple slices based on multiple vectorvalues respectively, and the method further comprises: sorting themultiple slices based on their respective MAPEVS.